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” . the  effort  to  produce  accommodation  (focus)  usually  results  in  some  degree  of  con¬ 

vergence  (depth  perception),  even  when  one  eye  is  covefed  -  the  so-called  accommodation- 
convergence-synkinesis.  It  exists  even  in  people  who  never  had  binocular  vision . " 

Gerald  Westheimer,  The  Eye.  [17] 


Abstract 

Rough  surfaces  pose  a  challenging  shape  extraction  problem.  Images  of  rough  sur¬ 
faces  are  often  characterized  by  high  frequency  intensity  variations,  and  it  is  difficult  to 
perceive  the  shapes  of  these  surfaces  from  their  images.  The  shape-from-focus  method  de¬ 
scribed  in  this  paper  uses  different  focus  levels  to  obtain  a  sequence  or  object  images.  The 
sum-modified-Laplacian  (SML)  operator  is  developed  to  compute  local  measures  of  the  qual¬ 
ity  of  image  focus.  The  SML  operator  is  applied  to  the  image  sequence,  and  the  set  of  focus 
measures  obtained  ai  each  image  point  arc  used  to  compute  local  depth  estimates.  We  present 
two  algorithms  for  depth  estimation.  The  first  algorithm  simply  looks  for  the  focus  level 
that  maximizes  the  focus  measure  at  each  point.  The  other  algorithm  models  the  SML  focus 
measure  variations  at  each  point  as  a  Gaussian  distribution  and  use  this  model  to  interpolate 
the  computed  focus  measures  to  obtain  more  accurate  depth  estimates.  The  algorithms  were 
implemented  and  tested  using  surfaces  of  different  roughness  and  reflectance  properties.  We 
conclude  with  a  brief  discussion  on  how  the  proposed  method  can  be  applied  to  smooth 
textuied  and  smooth  non-textured  surfaces,  f 

'  i' 
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1  Introduction 


1.1  Motivation 

•  The  advancement  of  three-dimensional  machine  vision  is  largely  dependent  on  the  develop¬ 
ment  of  efficient  and  reliable  shape  extraction  methods.  Shape  extraction,  in  turn,  requires 

#  a  sound  understanding  of  various  surface  reflection  mechanisms  and  the  image  formation 
process.  Many  extraction  methods,  for  diffuse  and  specular  surfaces,  have  been  developed 
in  the  past.  However,  the  extraction  problem  associated  with  rough  surfaces  has  not  received 
sufficient  attention.  All  surfaces  encountered  in  practice  are  rough  at  some  level  detail.  At 
that  level,  they  exhibit  high  frequency  spatial  surface  variations  that  are  often  random  in  na¬ 
ture.  In  many  vision  applications,  the  spatial  surface  variations  are  comparable  in  dimensions 
to  the  viewing  area  of  individual  picture  elements  of  the  imaging  sensor.  Hence,  image  in¬ 
tensities  produced  by  such  surfaces  vary  in  an  unpredictable  manner  from  one  sensor  element 
to  the  next,  and  it  is  difficult  tc  obtain  dense  and  accurate  surface  shape  information  by  using 
existing  techniques,  such  as,  structured  light,  shape-from-shadmg,  stereopsis,  etc.  Therefore, 
a  practical  and  reliable  solution  to  this  rather  difficult  extraction  problem  is  desirable. 


1.2  Background 

We  propose  to  use  focus  analysis  to  recover  the  shape  of  surfaces.  Previously,  focus  analysis 
has  been  used  to  automatically  focus  imaging  systems  and  obtain  sparse  depth  information 
from  the  observed  scene.  Horn  (1]  proposed  focusing  imaging  systems  by  using  the  Fourier 
transform  and  analyzing  the  frequency  content  in  the  image.  Tenebaum  [2]  developed  the 
gradient  magnitude  maximization  method  that  uses  the  sharpness  of  edges  to  optimize  focus 
quality.  Jarvis  [3]  proposed  the  sum-modulus-difference  that  is  computed  by  summing  the 
first  intensity  differences  between  neighboring  pixels  along  a  scan-line  and  is  used  as  a 
measure  of  focus  quality.  Schlag  et.al.  [4]  implemented  and  tested  various  automatically 
focusing  algorithms. 

Mere  recently,  Krotkov  [5][6]  evaluated  and  compared  the  performance  of  different 
focus  criterion  functions.  Krotkov  also  proposed  a  method  to  estimate  the  depth  of  an  image 
area.  Pentland  (7]  suggested  the  evaluation  of  image  blur  to  determine  the  depth  of  image 
points.  Grossmann  [8]  has  proposed  the  estimation  of  depth  of  edge  points  by  analyzing  the 
blur  of  the  edges  due  to  defocusing.  Darrell  and  Wohn  [9]  have  developed  a  depth  from 
focus  method  that  obtains  an  image  sequence  by  varying  the  focus  level  and  uses  Laplacian 
and  Gaussian  pyramids  to  calculate  depth.  Subbarao  [10]  suggests  the  change  of  fr-rinsic 
camera  parameters  to  recover  the  depth  map  of  a  scene.  Ohta  et.al.  [11]  and  Kaneda  et.al. 
[12]  have  used  images  corresponding  to  different  focus  levels  to  obtain  a  single  level  of  high 
focus  quality. 
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1.3  Proposed  Approach 


In  this  paper,  wc  develop  a  shape-from-focus  method.  We  start  by  defining  visibly  rough 
surfaces  that  produce  textured  images  with  high  frequency  intensity  variations.  We  review 
the  image  formation  process  and  show  that  a  defocused  imaging  system  plays  the  role  of 
a  low-pass  filter.  The  shape-from-focus  method  moves  the  unknown  object  with  respect  to 
the  imaging  system  and  obtains  a  sequence  of  images  that  correspond  to  different  levels  of 
object  focus.  The  sum-modified-Laplacian  (SML)  focus  operator  is  developed  to  measure 
the  relative  degree  of  focus  between  images.  The  operator  is  applied  to  the  image  sequence 
to  obtain  a  set  of  focus  measures  at  each  image  point.  The  focus  measure  values  at  each 
point  are  modeled  and  interpolated  to  obtain  accurate  depth  estimates.  Experimental  results 
indicate  that  the  method  is  capable  of  extracting  dense  and  accurate  shape  information  with 
appreciable  invariance  to  texture  strength  and  type. 

Though  the  shape-from-focus  method  is  developed  with  the  intention  of  extracting  the 
shape  of  rough  surfaces,  it  can  also  be  applied  to  smooth  surfaces  that  are  textured  as  a  result 
of  variations  in  surface  reflectance  properties.  Smooth  surfaces  that  do  not  have  textures  can 
also  be  handled  by  illuminating  these  surfaces  with  high  resolution  light  patterns  to  produce 
textured  images. 


2  Visibly  Rough  Surfaces 

In  the  study  of  reflection,  a  rough  surface  is  defined  as  one  whose  smallest  spatial  variations 
have  dimensions  that  are  much  larger  than  the  wavelength  of  the  incident  electromagnetic 
wave.  This  is  the  concept  of  optical  roughness.  In  this  paper,  we  introduce  the  notion 
of  visible  roughness ;  a  surface  is  considered  to  be  rough  if  the  dimensions  of  its  spatial 
variations  are  comparable  to  the  viewing  area  of  individual  elements  (e.g.  pixels)  of  the 
sensor  (e.g.  camera)  used  to  observe  the  surface.  The  surface  shown  in  Fig.l  is  composed 
of  a  large  number  of  facets'.  While  the  surface  appears  to  have  a  smoothly  varying  global 
shape,  z(x ,  y),  the  orientation  a  of  individual  facets  may  deviate  substantially  from  the  mean 
surface  orientation  in  the  facet  vicinity.  Although  the  facet  orientations  are  dependent  on  the 
global  shape  of  the  surface  and  on  the  orientations  of  neighboring  facets,  they  may  exhibit 
some  degree  of  randomness. 

Now  let  us  consider  the  images  of  rough  surfaces  generated  by  using  a  finite  resolution 
sensor.  The  number  of  facets  that  contribute  to  the  image  irradiance  at  a  pixel  location 
depends  on  the  magnification  of  the  optics  used  to  project  the  surface  onto  the  image  plane 
of  the  sensor.  We  define  two  levels  of  magnification;  multi-facet  level  and  facet  level.  At  the 

'No  assumptions  arc  made  regarding  the  sue  of  the  facets.  Hence,  these  facets  may  or  may  not  represent 
the  rmcro-facett  defined  in  [13],  (14). 


Figure  1:  Rough  Surface. 

multi-facet  level,  the  pixel  width  w  is  very  large  compared  to  the  facet  size  wy  (i.e.  w  =  vvy). 
In  this  case,  the  surface  patch  projected  on  a  pixel  may  be  modeled  by  assigning  it  a  mean 
orientation  value  and  a  roughness  which  is  determined  by  the  probability  function  of  its  facet 
orientations  [13]  [14].  The  pixel  intensities  are  continuous  functions  of  the  angle  of  incident 
light  and  can  be  expressed  as  a  linear  combination  of  the  diffuse  lobe  and  specular  lobe 
components  [14],  the  relative  strengths  of  the  two  components  depending  on  the  reflectance 
properties  of  the  facets. 

At  facet  level  magnification,  on  the  other  hand,  the  pixel  width  w  is  comparable  to 
the  facet  width  wy  (t.e.  w  =  w2),  and  only  one  or  few  facets  are  viewed  by  each  pixel.  As  a 
result  of  the  random  nature  of  facet  orientations,  the  image  intensity  values  are  expected  to 
vary  substantially  and  unpredictably  from  one  pixel  to  the  next.  This  is  true  for  both  specular 
as  well  as  diffuse  facets  as  the  radiance  of  both  are  dependent  on  the  angle  of  incident  light. 
Therefore,  at  the  facet  level,  the  surface  produces  images  that  are  rich  in  texture2-  and  we 
say  that  the  surface  is  visibly  rough.  But,  why  do  we  use  facet  level  measurements  when 
multi-facet  level  measurements  will  provide  us  with  image  intensities  that  can  perhaps  be 
used  to  extract  shape  information?  In  many  practical  instances,  the  desirable  resolution  of 
shape  information  is  unobtainable  at  the  multi-facet  leveL 

2There  are  many  noUons  of  what  is  meant  by  the  term  texture.  Here,  we  define  texture  as  a  noticeable 
fluctuation  in  the  intensities  of  neighboring  image  pixels  il6]  The  textures  produced  by  rough  surfaces  may 
be  periodic,  nearly  periodic,  or  random.  No  assumptions  are  made  regarding  the  type  of  texture. 


3  Focused  and  Defocused  Images 


In  this  section,  we  briefly  review  the  image  formation  process  and  describe  defocused  images 
as  processed  versions  of  focused  images.  There  are  two  approaches  to  the  study  of  optics, 
and  hence,  also  to  the  analysis  of  the  image  formation  process.  The  physical  optics  approach 
is  based  on  electromagnetic  wave  theory  [6],  and  it  analyzes  diffraction  effects  to  derive 
an  exact  image  formation  model.  Geometrical  optics ,  on  the  other  hand,  uses  the  short 
wavelength  of  light  to  simplify  the  analysis,  and  the  resulting  image  formation  model  may 
be  viewed  as  an  approximation  to  the  corresponding  physical  optics  model.  Since  our  sensor 
(i.e.  CCD  camera)  lacks  sufficient  spatial  resolution  to  make  diffraction  effects  significant, 
we  confine  ourselves  to  the  geometrical  analysis. 

Fig. 2  shows  the  basic  image  formation  geometry.  All  light  rays  that  are  radiated  by 
the  object  P  and  intercepted  by  the  lens  are  refracted  by  the  lens  to  converge  at  the  point 
Q  on  the  image  plane.  The  relationship  between  the  object  distance  o,  focal  distance  of  the 
lens  /,  and  the  image  distance  i,  is  given  by  the  Gaussian  lens  law: 


Each  point  on  the  object  plane  is  projected  onto  a  single  point  on  the  image  plane,  thus 
causing  a  clear  or  focused  image  If.x,y)  to  be  formed  on  the  image  plane.  If,  however,  the 
sensor  plane  does  not  coincide  with  the  image  plane  and  is  displaced  from  the  image  plane 
by  a  distance  6,  the  energy  received  from  the  object  by  the  lens  is  uniformly  distributed  over 
a  circular3  patch  on  the  senior  plane.  Fig. 2  may  be  used  to  establish  the  relationship  between 
the  radius  r  of  the  circular  patch  and  the  sensor  displacement  S.  By  using  similar  triangles, 
we  find  that: 

SR 

r  =  —  (2) 

i 

where  R  is  the  radius  of  the  lens.  It  is  also  possible  to  convince  oneself  that  the  radius  r  of 
the  circular  patch  is  independent  of  P's  location  on  the  object  plane.  Therefore,  the  blurred 
or  defocused  image  Id(x.  y)  formed  on  the  sensor  plane  can  also  be  obtained  by  convolving 
the  focused  image  Ifx,  y)  with  a  circular  symmetric  function  p(x,  y)  of  unit  volume,  called 
the  "pillbox"  function: 


where: 


Id(x,  y) 


pCx.y)  =  < 


=  p{x,  y)  *  I/ix,  y) 

(3) 

ih  if*2 +  s  <*> 

0  otherwise 

(4) 

3  me  shape  of  the  patch  also  depends  on  the  shape  of  the  aperture  of  the  imaging  system.  We  are  assuming 
the  aperture  to  be  circular. 


Figure  2:  Formation  of  focused  and  defocused  images. 


Note  that  the  defocusing  effect  is  observed  for  both  positive  and  negative  sensor  displace¬ 
ments 

Now  let  us  analyze  the  defocusing  process  in  the  frequency  domain  (u.  v).  If  lF{u.  vi. 
P(u.v ),  and  Ip(u.v)  are  the  Fourier  transforms  of  l/{x,  y),  p(x,  y),  and  fd(x.  y),  respectively, 
we  can  express  ecj.3  as: 

lD(u.\)  =  P(u,v).If(u-  v) 


The  pillbox  function  pCr.y)  is  a  rotationally  symmetric  function,  a' 
transform  of  rotationally  symmetric  functions  can  be  computed  as  a  single  in’ 
a  double  integral  by  using  the  Hankel  transform.  If  we  define  p  =  Vu2  +  v2.  - 
P(u.v)  in  polar  coordinates  as: 

*/»  =  ^ 

(rp) 


ne  Founer 
instead  of 
an  express 

(6) 


where  Jj(rp)  is  the  first-order  Bessel  function.  The  quantity  J,{rp)Krp)  plays  the  same  role  in 
two  dimensions  as  the  '  sine”  function  does  in  one  dimension.  P(p)  is  a  rotationally  symmetric 
function  of  which  a  one-dimensional  section  is  shown  in  Fig. 3.  P{p)  allows  low  frequencies 
of  I p{u,  v)  to  be  passed  unattenuated  while  it  attenuates  higher  frequencies.  Hence,  we  see 
that  defocusing  is  a  low-pass  filtering  process. 
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Figure  3:  One-dimensional  section  of  P(p). 

Due  to  the  sharp  cut-off  of  p(x.y),  ~P{p)  oscillates  about  zero  and  the  amplitude  of 
oscillation  decreases  asymptotically  as  {rp)~312 .  The  first  zero  of  P{p)  occurs  at  rp  =  3.83171. 

Therefore,  as  the  sensor  displacement  t  increases,  the  defocusing  radius  r  increases,  and  a 
smaller  range  of  low  frequencies  is  contained  in  the  defocused  image. 

From  Fig. 2,  it  can  be  seen  that  a  defocused  image  of  the  object  can  be  obtained  in 
three  ways:  by  displacing  the  sensor  with  respect  to  the  image  plane,  by  moving  the  lens,  or 
by  moving  the  object  with  respect  to  the  object  plane.  Moving  the  lens  or  sensor  plane  with 
respect  to  one  another  causes  the  following  problems: 

•  The  magnification  of  the  system  varies,  thereby  causing  the  image  coordinates  of  the 
object  points  to  change. 

•  The  area  on  the  sensor  plane  over  which  light  energy  is  distributed  varies,  thereby  * 

causing  a  variation  in  image  brightness. 

• 

In  order  to  overcome  these  problems,  we  propose  to  vary  the  degree  of  focus  by 
moving  the  object4  with  respect  to  a  fixed  configuration  of  the  optica]  system  and  sensor. 

‘Object  movement  is  easily  realized  in  industnal  inspection  applications. 
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This  approach  ensures  that  the  focused  areas  of  the  image  are  always  subjected  to  the  same 
magnification. 


4  Shape  from  Focus:  An  Overview 

The  shape-from-focus  method  is  based  on  the  observations  made  in  the  previous  sections. 

•  At  the  facet  level  magnification,  rough  surfaces  produce  images  i:  at  are  rich  in  texture. 

•  A  defocused  optical  system  plays  the  role  of  a  low-pass  filter 

Fig. 4  shows  a  rough  surface  of  unknown  shape  placed  on  a  rcnslational  stage.  The 
reference  plane  shown  corresponds  to  the  initial  position  of  the  stage.  The  configuration  of  the 
optics  and  sensor  define  a  single  plane,  the  "focused5  plane."  that  is  perfectly  focused  onto  the 
sensor  plane.  The  distance  dj  between  the  focused  and  reference  planes,  and  the  displacement 
d  of  the  stage  with  respect  to  the  reference  plane,  are  always  known  by  measurement.  Let 
us  focus  our  attention  on  the  surface  element,  s,  that  lies  on  the  unknown  surface,  S  If  the 
stage  is  moved  towards  the  focused  plane,  the  image  will  gradually  increase  in  its  degree 
of  focus  (high  frequency  content)  and  will  be  perfectly  focused  when  s  lies  on  the  focused 
plane  Further  movement  of  the  element  s  will  again  increase  the  defocusing  of  its  image 
If  we  observe  the  image  area  corresponding  to  the  s  and  record  the  stage  displacement  d  =  d 
a'  the  instant  of  maximum  focus,  we  can  compute  the  height  d,  of  s  with  respect  to  the  stage 
as  d,  =  df  -  d.  In  fact,  we  can  use  the  value  of  5  to  determine  the  distance  of  s  with  respect 
to  the  focused  plane,  sensor  plane,  or  any  other  coordinate  system  defined  with  respect  to 
the  imaging  system.  This  approach  may  be  applied  independently  to  all  surface  elements  to 
obtain  the  shape  of  entire  surface  5. 

To  automatically  detect  the  instant  of  "best"  focus,  we  will  develop  an  image  focus 
measure.  In  the  above  discussion,  the  stage  motion  and  image  acquisition  were  assumed  to 
be  continuous  processes.  In  practice,  however,  it  is  not  feasible  to  acquire  and  process  such 
a  large  number  of  images  in  a  reasonable  amount  of  time.  Therefore,  we  obtain  only  a  finite 
number  of  images;  the  stage  is  moved  in  increments  of  Ad,  and  an  image  is  obtained  at  each 
stage  position  (d  =  n.Ad).  By  studying  the  behavior  of  the  focus  measure,  we  develop  an 
interpolation  method  that  uses  a  small  number  of  focus  measures  to  compute  accurate  depth 
estimates.  An  important  feature  of  the  method  is  its  local  nature;  the  depth  estimate  at  an 
image  point  is  computed  only  from  focus  measures  recorded  at  tha.  point.  Consequently,  the 
method  can  adapt  well  to  variations  in  texture  type  and  content  over  the  object  surface. 

5The  focused  plane  is  the  same  as  the  object  plane  defined  in  the  previous  section.  A  different  name  is  used 
here  as  the  object  does  not  necessarily  lie  on  the  focused  plane. 
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5  A  Focus  Measure  Operator 
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In  order  to  measure  the  quality  of  focus  in  a  "small"  image  area,  we  would  like  to  develop 
a  focus  measure  operator.  The  operator  must  respond  to  high  frequency  variations  in  image 
intensity,  and  ideally,  must  produce  maximum  response  when  the  image  area  is  perfectly 
focused.  The  high  frequency  content  of  an  image  area  can  be  determined  by  using  the  Fourier 
transform  and  analyzir.r  the  frequency  distribution.  However,  since  Fourier  transforms  are 
expensive  to  compute  without  special  purpose  hardware,  we  seek  an  alternative  method. 

A  few  focus  rr  c  asure  operators  have  been  proposed  and  used  in  the  past.  Generally, 
the  objective  has  beer,  o  find  an  operator  that  behaves  in  a  stable  and  robust  manner  over  a 
variety  of  images  such  as  images  of  outdoor  scenes,  text,  etc.  Such  an  approach  is  essential 
while  developing  automatically  focusing  imaging  systems  that  have  to  deal  with  "general” 
scenes.  Bearing  in  mind  that  we  are  dealing  with  textured  images,  we  develop  an  operator 
that  is  particularly  well-suited  to  such  images.  In  the  next  section,  we  will  evaluate  our 
operator  and  compare  it  with  other  ones  by  using  textured  image  samples. 

One  way  to  high-pass  filter  an  image  is  to  determine  its  second  derivative.  For 
two-dimensional  images,  the  Laplacian  is  very  often  used: 


A2 1 


&1  &1_ 

dx3  dy2 


(7) 


where  !(x ,  y)  is  the  image  intensity  at  the  point  (x,  y).  We  note  that  in  the  case  of  the  Laplacian 
the  second  derivatives  in  the  x  and  y  directions  can  have  opposite  signs  and  tend  to  cancel 
each  other.  An  example  of  such  an  instance  is  illustrated  m  Fig.5;  the  partial  derivatives 
are  equal  in  magnitude  but  opposing  in  sign,  i.e.  A2!  =  0.  In  the  case  of  textured  images, 
this  and  similar  instances  may  occur  frequently,  and  the  Laplacian  is  prone  to  ‘.'(-have  in  an 
unstable  manner.  We  overcome  this  problem  by  defir,  •  g  the  modified  Lapla:  .  -s: 


A2m1 


eft 

dx2 


(8) 


Note  that  the  modified  Laplacian  is  always  greater  or  equal  in  magnitude  t>  r  Laplacian. 
The  discrete  approximation  to  the  Laplacian  is  usually  a  3x3  operator.  In  ore  o  accommo¬ 
date  for  possible  variations  in  the  size  of  texture  elements,  we  compute  the  p  ..  derivaDves 
by  using  a  variable  spacing  (step)  between  the  pixels  used  to  compute  the  derivatives.  Hence, 
the  discrete  approximation  to  the  modified  Laplacian  is  computed  as: 


ML(x,y)  -  |  2/(x,  y)  -  I(x-step,y)  -  I(x  +  step,y )  |  +  (9) 

|  2/(x,  y)  -  f(x,  y-  step)  -  I(x ,  y  +  step)  (  (10) 
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Figure  5:  A  texture  instance  with  zero  Laplacian  value. 

Finally,  the  focus  measure  at  a  point  (ij)  is  computed  as  the  sum  of  the  modified  Laplacian, 
in  a  "small"  window  around  07),  that  are  greater  than  a  threshold  value: 

i+N  j+N 

F(i,j)  =  ]T  ML(x,  y)  for  ML{x.y)  >T,  (Hi 

x=i-N  y=j-N 

where,  the  parameter  N  determines  the  window  size  used  to  compute  the  focus  measure.  In 
contrast  to  automatically  focusing  methods,  we  typically  use  a  small  window  of  size  3x3  or 
5x5,  i.e.  N  =  1  or  N  =  2.  We  shall  refer  to  the  above  focus  measure  as  the  sum-modified- 
Laplacian  (SML). 


6  Evaluating  the  Focus  Measure 

We  evaluate  the  SML  focus  measure  in  three  stages.  First,  we  analyze  its  behavior  as  a 
function  of  the  distance  between  the  observed  surface  and  the  focused  plane.  Next,  we  see 
how  the  SML  measure  is  affected  by  the  selection  if  its  parameters  step  and  Tj.  Finally,  we 
compare  the  performance  of  the  SML  operator  with  other  operators  used  in  the  past. 

A  detailed  description  of  the  experimental  set-up  is  given  in  a  later  section.  In  the 
following  experiments,  texture  samples  are  attached  to  a  translational  stage  (Fig.4)  and  the 
distance,  from  each  sample  to  the  stage  is  known  by  measurement.  Images  of  the  samples 
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are  obtained  using  a  microscope  and  a  256x256  pixel  CCD  camera.  The  complete  imaging 
system  has  a  physical  resolution  of  approximately  1  pm  per  pixel  width. 

In  Fig. 6,  the  focus  measure  functions  of  two  samples  are  shown.  Sample  X  has  high 
texture  content  while  sample  Y  has  relatively  weaker  texture.  Both  samples  are  made  of 
a  paste  containing  resin  and  tungsten  particles.  The  variable  size  of  the  tungsten  particles 
gives  the  surfaces  a  randomly  textured  appearance.  For  each  sample,  the  stage  is  moved 
in  increments  (Ad)  of  l  pm,  an  image  of  the  sample  is  obtained,  and  the  focus  measure 
is  computed  using  an  evaluation  window  size  of  lOx  10  pixels.  The  vertical  lines  in  Fig.6 
indicate  the  known  initial  distances  ( df  -  ds)  of  the  samples  from  the  focused  plane.  The  focus 
measures  were  computed  using  parameter  values  of  step  =  1  and  Tt  =  7.  No  form  of  temporal 
filtering  was  used  to  reduce  the  effects  of  image  noise,  as  we  intend  to  use  unfiltered  focus 
measures  to  estimate  the  depth  of  surface  points.  Though  the  measure  values  are  slightly 
noisy,  they  peak  very  close  to  the  expected  peak  positions  (vertical  lines  in  Fig.6).  We  see 
that,  the  focus  measure  function  peaks  sharply  for  the  stronger  texture  and  it  peaks  relatively 
slowly  and  with  a  lower  peak  value  for  the  weaker  texture.  However,  the  sharpness  of 
the  focus  measure  function  depends  not  only  on  the  texture  strength  but  also  the  "depth  of 
focus”  of  the  imaging  system.  The  depth  of  focus,  in  turn,  depends  on  the  magnification  and 
aperture  size  of  the  imaging  system.  We  will  assume  that  the  depth  of  focus  is  constant  for 
all  our  experiments. 

Fig.  7  shows  the  focus  measure  computed  as  a  function  of  the  parameter  step,  for 
the  sample  X  shown  in  Fig.6.  Once  again,  an  evaluation  window  size  of  10x10  pixels  and 
a  threshold  value  of  7/  =  7  were  used.  We  see  that,  for  sample  X,  a  maximum  measure 
value  is  computed  at  step  =  4.  Flowever,  it  may  be  noted  that  the  effective  size  of  the  focus 
measure  evaluation  window  increases  with  the  step  size.  As  we  are  interested  in  local  depth 
estimates,  step  values  of  I  or  2  are  generally  used.  Fig.8  shows  the  effect  of  varying  the 
threshold  7/,  for  both  focused  and  defocused  images  of  sample  X.  A  good  value  of  7/  is 
one  that  produces  a  high  measure  value  for  the  focused  image  and  low  measure  values  for 
defocused  images.  From  Fig.8  we  see  that,  for  sample  X,  7/  =  7  appears  to  be  a  good  choice. 
However,  from  a  number  of  unreported  experiments,  we  find  that  though  the  peak  value  of 
the  focus  measure  function  Fid)  varies  with  the  parameter  values,  the  same  parameter  values 
may  be  used  to  obtain  sharp,  unimodal,  focus  measure  peaks  for  surface  elements  of  varying 
degrees  of  texture  content. 

In  Fig.  10,  we  compare  the  SML  focus  measure  with  three  other  popular  measures: 
the  Tenem>rad,  variance ,  and  sum-Laplacian  (SL).  All  of  these  measures  are  defined  and 
evaluated  by  Krotkov  [5).  Here,  we  use  four  10x10  sample  windows  to  compare  the  mea- 
arcs.  The  differences  between  the  four  samples  are  illustrated  in  Fug. 9.  Sample  A  lies  on 
the  edge  of  a  textured  surface:  only  a  part  of  the  window  has  strong  texture.  Sample  B  lies 
’  :  •!  insaJe  the  edge  of  the  texture.  Sample  C  lies  well  within  edge  and  sample  D  lies  on  a 
•'triace  patch  that  is  oriented  away  from  the  viewing  direction  of  the  camera.  Real  samples 


Figure  6:  SML  focus  measure  function  computed  for  two  texture  samples. 
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Figure  7:  SML  focus  measure  as  a  function  of  the  operator  parameter  step 
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Figure  8:  SML  focus  measure  as  a  function  of  the  operator  parameter  T, 
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Figure  9:  Texture  samples  used  to  compare  focus  measure  operators. 

and  images  were  used  to  conduct  the  experiments.  Since  the  comparison  graphs  are  meant 
to  project  the  global  behaviors  of  the  operators,  they  are  plotted  using  large  incremental 
displacements  of  the  stage  {Ad  =  10/im).  Therefore,  the  actual  peaks  in  the  focus  measure 
function  can  lie  between  two  computed  measure  values.  The  measure  values  produced  by 
each  operator  are  normalized  by  the  maximum  measure  value.  Once  again,  the  vertical  line 
in  each  graph  indicate  the  known  depth  of  each  sample. 

The  Tenengrad  was  computed  using  a  threshold  value[5]  of  T  =  1,  and  the  SL  and 
SML  were  computed  using  Tj  =  7  and  step  -  1.  We  find  that  the  SL  and  SML  measures  peak 
sharply  and  close  to  the  expected  peak  positions  (vertical  lines).  However,  for  lower  measure 
values,  the  SL  is  more  noisy  than  the  SML.  For  weaker  textures  than  the  ones  used  above, 
this  causes  multimodal  measure  functions.  The  Tenengrad  and  variance  operators  produce 
relatively  weaker  modes  for  sample  A  as  both  operators  respond  to  first-order  variations  in 
image  intensity.  When  the  texture  edge  in  sample  A  is  defocused,  the  first-order  variations 
are  strong  although  the  second-order  variations  are  not.  As  a  result,  both  these  operators 
produce  high  measure  values  while  the  SL  and  SML  operators  do  not.  Sample  B  represents 
a  particularly  interesting  case.  When  the  edge  of  the  textured  surface  is  highly  defocused, 
it  is  also  subjected  to  a  different  magnification  than  when  it  is  in  focus.  Hence,  the  edge 
coordinates  change  and  the  image  of  sample  A  shifts  to  the  image  area  of  sample  B.  Since 
even  the  highly  defocused  image  of  the  edge  has  substantial  first-order  intensity  variations, 
the  Tenengrad  and  variance  operators  produce  high  measure  values.  This  effect  is  rather 
significant  when  the  mean  intensities  of  the  textured  and  background  areas  are  very  different. 
In  such  cases,  the  Tenengrad  and  variance  operators  produce  bimodal  focus  measure  functions 
that  result  in  erroneous  depth  estimates.  From  these  experiments,  we  conclude  that  the  SML 
operator  is  best  suited  for  measuring  the  focus  quality  of  textured  images. 
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Figure  10:  Comparison  between  Tenegrad,  variance,  SL,  and  SML  operators. 
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7  Sampling  the  Focus  Measure  Function 

We  can  represent  the  focus  measure  function  of  an  image  point  (jc,  y)  as  F(x,y,d).  Since  depth 

estimation  is  a  local  operation,  we  will  focus  our  attention  on  a  single  image  point,  bearing 

in  mind  that  the  same  estimation  method  can  be  applied  to  all  other  image  points.  Therefore,  » 

we  will  denote  the  focus  measure  function  as  simply  F{d).  By  studying  the  variation  of  F{d) 

in  Fig. 6,  we  find  that  F(d)  may  be  assumed  to  have  a  Gaussian  distribution  with  mean  value 

d  and  standard  deviation  of  (Fig.  11).  The  mean  5  corresponds  to  the  stage  displacement  at 

which  F(d)  is  maximum,  i.e.  F(d)  =  Fptak.  As  the  texture  content  of  the  surface  element 

increases,  Fpuk  increases  and  aF  decreases.  Each  surface  element,  therefore,  is  expected  to 

have  its  own  Fplak  and  of  values. 


F 


Figure  11:  Sampling  the  SML  focus  measure  function. 

If  we  use  very  small  stage  displacement  ( Ad  0),  the  number  of  images  to  be  ob-  • 

tained  and  processed  is  too  large  from  the  perspective  of  practical  implementation.  Hence, 
we  use  large  displacements  to  obtain  a  few  images  of  different  focus  levels  and  use  the  « 

Gaussian  model  to  interpolate  the  small  number  of  focus  measures  to  obtain  depth  estimates 
at  each  image  point.  Computing  the  focus  measure  at  a  finite  number  of  displacements  is 
equivalent  to  sampling  the  function  F(d)  (Fig.  11);  at  each  displacement  d,  we  compute  the 
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focus  measure  Fid,)  to  obtain  the  set  {Fid,)  |  1,2,  ....M).  We  show  in  the  following  sec¬ 

tion  that  a  minimum  of  three  focus  measures  are  needed  perform  the  Gauss. an  interpolation. 
In  theory,  therefore,  depth  estimates  may  be  obtained  from  only  three  images  of  the  surface. 
However,  since  the  Gaussian  model  only  approximates  the  focus  measure  function,  we  use 
the  condition  aF  <  Ad  <  2oF  to  ensure  that  evaluation  of  at  least  one  focus  me  a  are  in  the 
±  aF  range  of  Fid).  Note  that  displacements  are  applied  to  all  object  points.  Therefore,  by 
applying  the  above  condition  to  the  image  point  that  has  maximum  texture  content,  we  can 
ensure  that  a  few  or  many  focus  measures  will  be  computed  in  the  ±  aF  range  at  all  image 
points. 

We  note  that  the  value  of  aF  also  increases  with  the  depth  of  focus  of  the  imaging 
system.  Therefore,  for  objects  of  larger  dimensions  also,  only  a  small  number  of  images 
may  used  by  increasing  the  depth  of  focus. 
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8  Depth  Estimates  from  Focus  Measures 

In  this  section,  we  describe  the  estimation  of  depth  of  a  surface  point  (at.  y)  from  the  focus 

measure  set  [Fid,)  |  i  =  1.2 . M).  We  use  the  parameter  d  to  represent  the  depth  of  the 

surface  point.  For  convenience,  the  notation  F,  is  used  to  represent  the  focus  measure  value 
F(d:).  We  present  algorithms  for  two  different  depth  estimation  methods.-  Each  algorithm 
may  be  applied  to  all  points  in  the  image  to  obtain  depth  maps. 

8.1  Coarse  Resolution  Depth  Estimation 

The  first  algorithm  simply  looks  for  the  displacement  value  d ,  that  maximizes  the  focus 
measure  and  assigns  that  value  to  d. 

Algorithm  1 

Step  1:  Let  k  =  1 ,  F^  =  0. 

Step  2:  If  Fk  >  F^,  F^  =  Fk  and  d  =  dk. 

Step  3:  If  k  <  M,  k  =  k  +  /,  go  to  step  2. 

Step  4:  If  Fw  <  T2,  the  point  (.x,  y)  belongs  to  background.  Stop. 

This  simple  algorithm  may  be  used  to  compute  rough  depth  estimates.  The  perfor¬ 
mance  of  the  algorithm  is  directly  dependent  on  the  selection  of  Ad. 
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8.2  Depth  Estimation  by  Gaussian  Interpolation 


The  second  algorithm  uses  the  Gaussian  distribution  to  model  the  focus  measure  function 
F(d)  and  interpolates  the  computed  measure  values  to  obtain  more  accurate  depth  estimates. 
One  approach  is  to  fit  all  computed  F,  values  to  the  Gaussian  model.  However,  we  feel 
that  more  accurate  depth  estimates  can  be  obtained,  while  saving  computations,  by  using  the 
Gaussian  distribution  to  model  only  the  peak  of  F(d).  The  following  algorithm  uses  only 
three  focus  measures,  namely,  Fm_h  Fm ,  and  F^j,  that  lie  on  the  largest  mode  of  F(d),  such 
that,  Fm  >  Fm_,  and  Fm  >  Fm+,  (Fig.12). 

Using  the  Gaussian  model,  the  focus  measure  function  may  be  expressed  as: 

f  =  }  (12. 


where  dand  aF  are  the  mean  and  standard  deviation  of  the  Gaussian  distribution  (Fig.  15j. 
Using  natural  logarithm,  we  can  rewrite  Eq.  12  as: 

InF  =  lnFptak  -  ^  <l3> 


By  substituting  each  of  the  three  measures  Fm_j,  Fm ,  and  Fm+;,  and  its  corresponding 
displacement  value  in  Eq.  13,  we  obtain  three  equations  that  can  be  solved  for  d  and  aF: 

d  =  (-lnF'*  ~  I”Fm,i)(dm2  -  dm_/)  -  (lnFm  -  lnFm_,)(dm2  -  dm.2) 
2Ad{(lnFm  -  lnFm .,)  +  (lnFm  -  lnFm. ,)} 

2  =  (■ dm 2  -  dn-h  +  {dj  -  dm.,2) 

°F  2{(lnFm  -  lnFm .,)  +  (lnFm  -  lnFm. >,)} 

Using  Eq.  12,  we  can  find  Fpeak  from  oF  and  d  as: 

Fptot  =  Fm  j  exp  j-  J  06) 


If  Fptak  is  large  and  aF  is  small,  the  focus  measure  function  has  a  "strong"  peak,  indicating 
high  surface  texture  content  in  the  vicinity  of  the  image  point  (x,  y).  Thus,  the  values  of  Fpcak 
and  Of  can  be  used  to  segment  the  observed  scene  into  regions  of  different  texture  content. 

The  following  algorithm  first  finds  the  measures  Fm_/(  Fm,  and  Fm+/  that  correspond 
to  the  strongest6  peak  of  F{d)  and  then  uses  these  measures  to  estimate  the  depth  d  by  Gaus¬ 
sian  interpolation. 

6Due  to  image  noise  and  variations  in  magnification,  the  focus  measure  function  may  be  multi-modal  with 
one  strong  peak  and  one  or  more  weak  ones. 
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Figure  12:  Gaussian  interpolation  of  focus  measures. 


Algorithm  2 


Step  1:  Let  k  -  3,  Fm_t  =  0,  Fm  =  0,  Fm. ,  =  0,  dm  =  0. 
Step  2:  If  Ft_i  >  Fm  ,  Fk.t  >  Fk  ,  and  Fk.i  >  Fk,2l  then. 


Fm  =  Fk_h 
Fm-J  =  Fk .2, 

F  1  =  Fk, 

dm  =  dk_i. 

Step  3:  If  k  <  M,  k  =  k  +  1,  go  to  step  2. 

Step  4:  dm_,  =  dm  -  Ad  and  dm.,  -  dm  +  Ad.  Determine  3,  c rF,  and  Fpm)k  using  Eqs.  14, 
15,  and  16. 

•  Step  5:  If  Fptak  <  T3  or  oF  >  T4,  the  image  point  ( x,y )  belongs  to  background.  Stop. 

Since  the  values  of  Fptak  and  rrF  are  only  useful  for  texture  classification,  their  eval¬ 
uation  may  be  avoided  to  save  computations. 


19 


9  Experiments 

9.1  Experimental  Set-up 

Fig.  13  shows  a  photograph  of  the  experimental  set-up  used  to  demonstrate  the  shape-from-  % 

focus  method.  A  schematic  diagram  of  the  set-up  is  shown  in  Fig.  14.  A  microscope  is  used 
to  magnify  the  object  surface,  and  images  are  obtained  using  a  CCD  camera  with  256x256 
pixels.  Camera  images  are  digitized  and  processes  using  a  computer.  The  magnification  of 
the  imaging  system  can  be  varied  from  x5  to  xl60.  The  object  is  placed  on  a  translational 
stage  that  is  used  to  move  the  object  through  the  focused  plane  of  the  imaging  system.  Stage 
displacements  are  monitored  using  an  electronic  displacement  sensor  that  has  an  accuracy 
of  within  0.1  /j m.  In  most  of  our  experiments,  the  bright  field  illumination  (Fig.  14)  of  the 
microscope  was  used  to  illuminate  the  object  surface. 

9.2  Results 

The  accuracy  of  the  shape-from-focus  algorithms  was  analyzed  using  a  steel  ball  sample 
that  was  1590/i-m  in  diameter.  The  ball  was  sand-papered  to  give  it  a  rough  surface  A 
camera  image  of  the  ball,  under  bright  field  illumination,  is  shown  in  Fig. 15a.  Due  to  the 
small  depth  of  focus  of  the  microscope,  some  areas  of  the  ball  are  defocused.  The  bright 
field  illumination  decreases  the  texture  intensity  from  the  flat  top  section  of  the  ball  to  the 
steep  boundary  area.  As  it  is  difficult  to  perceive  the  shape  of  most  of  the  samples  we  have 
used  from  their  camera  images,  we  have  also  included  scanning  electron  microscope  (SEMj 
images  of  the  samples.  We  hope  that  these  images  will  provide  sufficient  shape  cues  to  the 
reader  An  SEM  image  of  the  ball  is  show  n  in  Fig.  1 5b.  Incremental  displacement  of  Ad  = 

100  f.m  were  used  to  obtain  1 3  images  of  the  bal,  and  a  5  x5  SML  operator  was  applied  to  the 
image  sequence  to  obtain  focus  measures.  Depth  maps  of  the  ball,  generated  by  the  coarse 
resolution  and  Gaussian  interpolation  algorithms,  are  shown  in  Fig. 15c  and  15d,  respectively. 

The  known  size  and  location  of  the  ball  were  used  to  obtain  error  maps  by  subtracting  a 

smooth  ball  from  the  two  depth  maps.  It  is  difficult  to  define  the  accuracy  of  the  method 

as  it  depends  on  many  factors:  the  surface  texture,  depth  of  focus  of  the  imaging  system, 

and  the  incremental  displacement  Ad.  The  table  shown  in  Fig.l5e  shows  the  error  statistics 

computed  from  the  error  maps  corresponding  to  the  two  algorithms.  A  total  of  23235  image 

pixels  lie  within  the  boundary  of  the  ball.  The  number  of  depth  values  computes  by  each 

algorithm  depends  on  the  selected  values  for  the  thresholds  7j,  and  T4.  The  error  map 

corresponding  to  the  Gaussian  interpolation  algorithm  is  shown  in  Fig.ISf.  We  see  that  there  • 

is  no  obvious  corre!  ition  between  the  errors  and  the  surface  orientation. 

Fig  16  to  19  show  samples  with  different  surface  reflectance  and  roughness  properties, 
and  their  depth  maps  obtained  using  the  Gaussian  interpolation  algorithm.  All  sample  are 
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approximately  100  pm  in  width  and  an  incremental  displacement  of  Ad  =  10  um  was  used 
in  all  cases  to  obtain  sequences  of  about  10  images  each.  Fig  16a  and  16b  show  the  camera 
and  SEM  images  oi  a  tungsten  paste  filling  in  a  via-hole  on  a  substrate  [15]  that  is  used 
to  establish  electrical  connections  between  different  components.  Conditions  such  as  excess 
filling  and  lack  of  filling  cause  electrical  defects  such  as  short  and  open  circuits.  The  sample 
shown  in  Fig.  16  has  a  bump  on  its  surface,  indicating  excess  filling.  The  specular  reflectance 
and  variable  size  of  the  tungsten  particles  gives  the  surface  a  random  texture.  The  white 
background  (Fig.  16a)  is  the  substrate  area  that  has  very  weak  texture.  For  this  sample,  we 
selected  the  threshold  values  to  classify  the  substrate  area  as  background.  An  arbitrary  depth 
value  is  assigned  to  the  background  region. 

Fig.  17  show's  another  via-hole  sample.  In  this  case,  the  substrate  and  filling  are 
hardened  by  baking.  The  baking  process  changes  the  reflectance  and  texture  of  the  filling 
and  also  increases  the  texture  content  of  the  substrate.  From  the  SEM  image  we  see  that  the 
filling  has  some  contamination  on  its  surface  and  a  concavity  in  the  center.  For  this  sample, 
the  algorithm  threshold  values  were  selected  to  obtain  the  depth  of  the  substrate  area  too. 
To  accommodate  for  the  large  size  of  substrate  texture  elements,  a  step  size  ot  2  was  used. 
Two  different  views  of  the  sample’s  depth  map  are  shown  in  Fig. 17c  and  d.  :  shows 

another  baked  via-hole  sample.  In  this  case,  the  the  via-hole  is  not  sufficient!  d  with 

tungsten  paste.  Again,  two  views  of  the  computed  depth  maps  are  shown. 

Fig. 19  shows  a  via-hole  filling  that  is  plated  with  Nickel.  The  plating  procv  .ncreases 
the  panicale  size  and  each  particle  is  highly  specular  in  reflectance.  Due  to  th-  phencal 
shape  of  the  panicles  (Fig.  19b),  the  bright  field  illumination  produces  a  single  :  at  the 
center  of  each  panicles  and  some  of  the  steep  surface  areas  do  not  reflect  ar  'ht.  To 

demonstrate  the  use  of  special  illumination,  we  used  a  multi-spot  light  source  .  uminate 

the  sample  and  obtained  images  of  higher  texture  content,  such  as  the  one  shown  in  Fig. 19c. 
The  computed  depth  map  of  the  sample  is  shown  in  Fig.l9d. 

9.3  Discussion 

The  above  expenments  validate  the  effectiveness  of  the  shape-from-focus  method.  The 
Gaussian  interpolation  algorithm  performs  stably  over  a  wide  range  of  textures  "ors  in 
computed  depth  estimates  result  from  factors,  such  as  image  rrise,  Gaussian  ap;  -nation 
of  the  SML  focus  measure  function,  and  weak  textures  in  so..:C  image  areas.  S  e  detail 
of  the  surface  roughness  is  lost  due  to  the  use  of  a  finite  size  window  to  compute  "  e  SML 
focus  measures. 

Most  of  the  above  experiments  were  conducted  using  bright  field  ilium:  n.  In 

the  last  experiment,  we  demonstrated  the  use  of  special  illumination  to  enhance  t  ualitv 

of  textured  images.  Furthermore,  light  patterns  can  be  projected  on  smooth  sun.  both 


21 


diffuse  and  specular, to  generate  textured  images.  Hence,  by  using  special  illumination,  the 
shape-from-focus  method  can  also  be  applied  to  smooth  surfaces. 

In  the  above  experiments,  object  displacement  and  image  aquisition  were  manually 
initiated.  In  a  high-speed  implementation,  the  object  can  be  moved  continuously,  while 
images  are  obtained  at  fixed  intervals  of  time.  By  using  customized  hardware,  the  SML 
focus  operator  can  be  applied  to  each  image  in  a  frame-time  and  the  Gaussian  interpolation 
can  be  implemented  by  using  look-up  tables.  We  estimate  that  a  high-speed  implementation  # 

of  the  method  can  generate  surface  depth  maps  in  less  than  1  second. 


10  Conclusion 

In  this  paper,  we  have  presented  shape-from-focus  as  a  new  method  of  extracting  the  shape 
of  rough  surfaces. 

•  To  measure  the  quality  of  image  focus,  we  developed  the  SML  operator.  By  evaluating 
the  SML  operator  and  comparing  it  to  other  existing  operators,  we  found  that  the  SML 
operator  is  particularly  well-suited  for  measuring  the  focus  of  textured  images. 

•  We  developed  and  tested  two  depth  estimation  algorithms  and  found,  through  numerous 
experiments,  that  the  Gaussian  interpolation  algorithm  produces  accurate  results  for  a 
variety  of  textures. 

•  The  local  nature  of  the  depth  estimation  technique  enables  it  to  adapt  to  substantial 
variations  in  image  texture. 

•  TTiough  we  have  concentrated  on  rough  surfaces  in  this  paper,  the  shape-from-focus 
method  can  be  directly  applied  to  smooth  textured  surfaces.  Smooth  non-textured 
surfaces  can  also  be  handled  by  using  special  illumination  techniques. 
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Diameter  of  Test  Sphere  :  1590  pm 


Mean  fcrror  (pi..) 


Mean  Absolute 
Error  (pm) 


Maximum  Absolute 
Error  (pm) 


(e)  Error  statistics. 


(f)  Error  map:  Gaussian  interpolation. 


Figure  15:  Steel  ball.  D  ont  imicd) 
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